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ENHANCEMENT OF COMPRESSED IMAGE DATA 

Cross Reference to Related U.S. Patents 

[0001] Cross reference is made to the following commonly assigned U.S. Patents 

or applications to Eschbach et al., each of which is hereby expressly incorporated by 
reference herein: U.S. Patent No. 5,414,538; U.S. Patent No. 5,357,352; U.S. Patent No. 
5,450,502; U.S. Patent No. 5,371,615; U.S. Patent No. 5,363,209; U.S. Patent No. 
5,450,217; and, U.S. Patent No. 5,802,214 and Serial number 09/448,947. Similarly, 
commonly assigned U.S. Patent No. 5,347,374 to Fuss et al. is also hereby expressly 
incorporated by reference herein. 

Background of the Invention 

[0002] In the past, a typical application for copiers or scan-to-print image 

processing systems was to reproduce an input image as accurately as possible, i.e., 
render a copy. Thus, copies have been rendered as accurately as possible, including 
flaws in the source image. However, as customers have become more sophisticated in 
their document reproduction requirements, they have recognized that an exact copy is 
often not desired. Instead, an inexact copy that is perceived as a higher quality image has 
been deemed more desirable. 

[0003] Until recently, image quality from the output of a copier or a scan-to-print 

system was directly related to the quality of the input image. While this has been 
satisfactory for reproducing images of text or line drawings, it has been found to be sub- 
optimal for purposes of reproducing photographs and other more complex images. With 
photographs, in particular, reproduction is complicated given the inexact nature of the 
discipline, variations in equipment, aging of photographs, and the like. Given that the 
input image is often poor quality, it has been deemed desirable to render an output image 
that is perceived to be the "best" possible image (or at least superior to the input image), 
rather than an exact copy. 



[0004] The above-noted patents to Eschbach et al. and Fuss et al. disclose an 

Automated Image Enhancement (AIE) system. This system receives an optionally sub- 
sampled description of the input image, and alters the tone reproduction curve (TRC) ~ 
a curve that defines the relationship of an input image signal to an output image signal 
for purposes of enhancement ~ and/or derives or alters a sharpness filter for that image 
on an image-by-image basis, as appropriate, so that the resulting output image is 
perceived to be superior to the input image. AIE is used, for example, to alter perceived 
exposure, luminance, sharpness, saturation, color balance, and the like to provide an 
output image that is preferable to the input image. It is important to note that AIE 
performs well even if it has to rely upon a sub-sampled image to statistically analyze the 
overall image. AIE does not require that each and every item of image information be 
analyzed. Of course, this sub-sampling speeds image enhancement operations and 
reduces the size/number of memory buffers required to implement the AIE system. 
[0005] More particularly, for example, U.S. Patent No. 5,414,538 entitled 

"Image-Dependent Exposure Enhancement" discloses a method of altering the perceived 
exposure of an output image produced from an input image that includes: (a) receiving 
the input image defined in terms of red-green-blue (RGB) signals; (b) converting the 
RGB signals to corresponding luminance-chrominance signals including at least one 
signal that represent overall image intensity; (c) comparing the intensity signal to upper 
and lower intensity threshold signals that define the acceptable levels of brightness and 
darkness in the image; (d) if one of the thresholds is exceeded, the image signal 
representative of image intensity is processed according to a select equation, and a TRC 
associated with the image is adjusted so that exposure characteristics of the resulting 
output image are perceived to be superior to those of the input image. 
[0006] U.S. Patent No. 5,450,502 entitled "Image-Dependent Luminance 

Enhancement" discloses a method of altering the perceived luminance of an output 
image produced from an input image that includes: (a) receiving the input image data 
defined in terms of a color space; (b) if required, converting the input image data into a 
luminance-chrominance color space wherein at least one term bears a relationship to 
overall intensity of the input image; (c) deriving a global intensity histogram for the 
overall input image; (d) filtering the histogram signal to flatten high peaks and low 
valleys without altering its relatively flat portions; and, (e) utilizing the filtered 
histogram signal to control TRC mapping in a device with which the image is to be 
rendered. In accordance with another aspect of the disclosed luminance enhancement 



method, the input image can be divided into plural regions, and a local intensity 
histogram signal can be derived for each region. If any of the local histogram signals are 
flatter than the global histogram signal, the local signals are summed and used in place 
of the global histogram as input to the histogram flattening filter. 
[0007] The methods described in the Eschbach et al. '538 and '502 patents, and 

the other Eschbach et al. and Fuss et al. patents noted above, are described in connection 
with uncompressed image data. However, in many image processing operations, the 
image data is retrieved from an image storage device or other location or is otherwise 
supplied in a compressed form to minimize image storage space. 

[0008] Thus, heretofore, image enhancement operations typically have required 

that the compressed images first be decompressed or "decoded" for image enhancement 
operations. This is undesirable in that the decompression operation slows the overall 
enhancement operation, the image processing apparatus must be provided with 
additional memory to accommodate large amounts of uncompressed image data, and 
each compression operation itself, further degrades the image data due to data loss. 
[0009] Accordingly, it is desirable to provide method and system which 

overcomes the above-mentioned difficulties and others. 

Summary of the Invention 

[0010] In accordance with one embodiment of the present invention, an image 

processing method includes receiving packets of compressed image data. A first subset 
of the compressed data is decompressed and an adjustment is determined from the first 
subset to be applied to the image data. 

[0011] In accordance with another aspect of the present invention, the method 

further includes continued decompressing of additional subsets resulting in a 
decompressed image at a desired resolution. The adjustment is then applied to the 
decompressed image. 

[0012] In accordance with another aspect of the present invention, the method 

further includes incorporating the adjustment into the compressed image data, and 
storing the compressed image data and the incorporated adjustment. 
[0013] In accordance with another embodiment of the present invention, an 

image processing method includes receiving wavelet domain data defining a digital 
image where the wavelet domain data includes N levels of decomposition. M levels, 
where M is less than N, are then input into an automated image enhancement system 



which derives at least one of a tone reproduction curve and a sharpness filter for 
enhancement of the input digital image data. The image data is more completely 
decompressed, either concurrently or at a later time, to obtain decompressed image data 
representing the digital image at a selected level of decomposition. Using either or both 
of the final reproduction curve or sharpness filter, the decompressed image data is 
enhanced. 

[0014] In accordance with another aspect of the present invention, the method 

further includes storing either or both the tone reproduction curve or sharpness filter for 
subsequent use in enhancing the digital image. 

[0015] In accordance with another aspect of the present invention, the step of 

inputting M levels into an automated image enhancement system includes inputting the 
M levels to at least one of an exposure enhancement module, a color balance 
enhancement module, a contrast enhancement module, and a luminance enhancement 
module. 

[0016] In accordance with another aspect of the present invention, the step of 

storing either or both of the final reproduction curve or the sharpness filter for 
subsequent use includes binding the tone reproduction curve or sharpness filter to the 
received image data. 

[0017] In accordance with another aspect of the present invention, the inputting 

M levels into an automated image enhancement system includes inputting the M levels 
to a sharpness enhancement module that derives a sharpness filter. 

[0018] In accordance with another embodiment of the present invention, a 

wavelet domain image processing method includes receiving image data defining an N 
level wavelet decomposition of a digital image. Each level n=l through N of said 
decomposition includes a horizontal subband H, a vertical subband V, and a diagonal 
subband D. Image enhancement analysis is then performed using a low pass subband on 
an Mth level wavelet decomposition, where M<N. 

[0019] In accordance with another embodiment of the present invention, a 

xerographic apparatus which processes wavelet domain image data includes means for 
receiving the wavelet domain image data representative of the input digital image. The 
wavelet domain image data includes N levels of wavelet decompositions. An M-level 
extractor extracts an Mth level wavelet decomposition from the wavelet domain image 
data where M<N. The extracted Mth level wavelet decomposition is a low resolution 
representation of the input digital image. An image enhancement system receives the 



extracted Mth level wavelet decomposition and derives an enhancement process from 
the extracted wavelet decomposition to adjust higher resolution representations of the 
digital input image. 

[0020] In accordance with another aspect of the present invention, the 

enhancement process includes a tone reproduction curve. 

[0021] In accordance with another aspect of the present invention, the 

enhancement process includes a sharpness filter. 

[0022] In accordance with another aspect of the present invention, the apparatus 

fixrther includes means for converting the wavelet domain image data into the high 
resolution representation of the input digital image and a processor which applies the 
enhancement process to the high resolution representation. 

[0023] In accordance with another aspect of the present invention, the apparatus 

further includes a storage which at least temporarily stores the enhancement process for 
subsequent use in enhancing the input digital image. 

[0024] In accordance with another aspect of the present invention, the storage 

includes data representative of the enhancement process inserted in the wavelet domain 
image data. 

[0025] In accordance with another aspect of the present invention, the image 

enhancement system includes at least one of an exposure enhancement module, a color 
balance enhancement module, a contrast enhancement module, and a luminance 
enhancement module. 

[0026] Still further advantages and benefits of the present invention will become 

apparent to those of ordinary skill in the art upon reading and understanding the 
following detailed description of the preferred embodiment. 

Brief Description of the Drawings 

[0027] The present invention may take form in various steps and arrangements 

of steps, and in certain components and arrangements of components. The drawings are 
only for purposes of illustrating the preferred embodiments and are not to be construed 
as limiting the invention. 

[0028] FIGURE 1 is a diagrammatic illustration of a digital image processing 

apparatus in accordance with the present invention; 

[0029] FIGURE 2 is a diagrammatic illustration of a single-level wavelet 

decomposition of a digital image; 



[0030] FIGURE 3 is a diagrammatic illustration of multiple-level (i.e., 3 level) 

wavelet decomposition of a digital image; 

[0031] FIGURE 4 is a diagrammatic illustration of a system which suitably 

practices one aspect of the present invention; 

[0032] FIGURE 5 is a diagrammatic illustration of a system which suitably 

practices another aspect of the present invention; 

[0033] FIGURE 6 is a diagrammatic illustration of a system which suitably 

practices the present invention according to the embodiment illustrated in Figure 5; and 
[0034] FIGURE 7 is a detailed diagrammatic illustration of an enhancement 

mechanism which suitably practices the present invention. 

Detailed Description of the Preferred Embodiment 

[0035] Referring now to the drawings wherein the showings are for purposes of 

illustrating a preferred embodiment of the invention only and not for limiting the 
invention in any way, FIGURE 1 diagrammatically illustrates an image processing 
apparatus including an image input terminal 12, an image processing unit (IPU) 14, and 
an image output terminal 16. The input terminal 12 can be any source of digital image 
data including a scanner , digital camera, or a source of stored image data. The image 
processing unit 14 is provided by any suitable electronic computing apparatus including 
a microprocessor or the like for carrying out digital image processing as described 
herein. Image data from the input terminal 12 are supplied to the IPU 14 for processing. 
After processing the data, the IPU 14 outputs the processed data to the image output 
terminal 16 which can include a non-impact printer, video display, image storage device 
or the like. 

[0036] The decomposition of a digital image into different frequency subbands 

or subband images using a discrete wavelet transform is known. The discrete wavelet 
transform and its inverse transform are well-known and fully described in Wavelets and 
Subband Coding, Vetterli and Kovacevic, Prentice Hall PTR (1995); Wavelets and Filter 
Banks, Strang and Nguyen, Wellesley-Cambridge Press (1996); and, Wavelet Basics, 
Chan, Kluwer Academic Publishers (1995), each of which is hereby expressly 
incorporated by reference herein. 

[0037] Referring now to FIGURE 2, a one-level wavelet decomposition 

corresponding to an original digital image is illustrated. As is known, the wavelet 
decomposition of a digital image comprises four subbands LL,LH,HL,HH. The low- 



pass subband LL is defined by a low-pass, sub-sampled version of the original image 
(i.e., by the low-frequency information for both the vertical and horizontal directions in 
the original image); the horizontal subband LH is defined by the low-frequency vertical 
information and the high-frequency horizontal information in the original image; the 
vertical subband HL is defined by the high-frequency vertical information and the low- 
frequency horizontal information in the original image; and the diagonal subband HH is 
defined by the high-frequency diagonal information in the original image (i.e., by the 
high-frequency information for both the vertical and horizontal directions in the original 
image). 

[0038] Referring now to FIGURE 3, the first level wavelet decomposition of the 

original image comprises a horizontal subband HO, a vertical subband V0 and a diagonal 
subband DO as described above. The low-pass subband, itself, has been subjected to a 
wavelet decomposition so that a second-level of subbands H1,V1,D1 are defined. Here, 
again, the second-level low-pass subband has been subjected to a wavelet decomposition 
so that a third level of wavelet subbands is defined by a third-level low-pass subband 
LL2, a third-level horizontal subband H2, a third-level vertical subband V2 and a third- 
level diagonal subband D2. The wavelet decomposition can be repeated further if 
desired. 

[0039] With reference now to Figure 4, wavelet compressed image data 30 is 

transmitted to IPU 14. When wavelet compressed image data is streamed, typically the 
most compressed subbands arrive first and increased resolution is obtained by 
subsequent bands. Conventionally, the number of decompositions is generically 
identified with the letter N, and specific decomposition levels are generically referred to 
with the letter M, where the 1<M<N. The IPU 14 receives the stream and decompresses 
image data to a desired resolution level M, 32. The relatively low resolution level M, is 
however suitable for an enhancement analysis and development of parameters useful to 
enhance the entire decompressed image. Accordingly, the IPU 14 performs an 
enhancement analysis of the image data in the low pass subband LL at the resolution 
level M, 34. In one embodiment, the image data continues to stream into the IPU 14 and 
this data is continually decompressed until all the levels have been compressed or until 
the decompression is stopped, 38. The decompressed image 40 and the enhancement 
parameters 42 are supplied to an image processing algorithm 44 which applies the 
enhancement parameters to the decompressed image. 



[0040] With reference now to Figure 5, wavelet compressed image data 30 

enters an IPU 14' and decompressed to resolution level M, 32. AIE analysis is then 
performed on the low pass subband LL of the resolution level M, 50. The AIE analysis 
corresponds to the previously mentioned patents. Results of the AIE analysis are 
provided to a data and parameter binder 52 which combines the compressed data stream 
with data representative of the enhancement determined from the AIE analysis. This 
bound data stream of data and enhancement parameters are then stored in an enhanced 
storage device 56 located in the IPU 14, on a server, in a printer and the like. 
[0041] Referring now to Figure 6, in an example of using bound data and 

enhancement parameters, the data is extracted from the enhanced storage 56 and an 
extraction algorithm in the IPU 14" extracts the enhancement analysis parameters 54 
from the data. The data continues decompression to a desired level 60, which when 
achieved is forwarded to an image processing algorithm 62 which applies the 
enhancement parameters 54 to the decompressed image. IPU 14" then outputs the 
enhanced decompressed image for rendering, storage and/or other further image 
processing or output, 66. 

[0042] Referring now to Figure 7, a more detailed illustration of the AIE analysis 

and processing is illustrated. JPEG 2000, for example, wavelet compressed image data 
30 is received, and an M-level extractor 32 extracts the low-pass subband LL of a 
determined compression level from the JPEG 2000 compressed data. The JPEG 2000 
compressed data stream is unaltered by the M-level extractor 32, and is input to the 
binder unit 52 by way of a data channel 100. 

[0043] The low-pass subband LL from the Mth-level is input via data channel 

102 to the automated image enhancement (AIE) unit 50. In particular, the low-pass 
subband LL from the Mth-level is placed into an information channel 108 that is, in 
turn, operatively connected to a plurality of AIE modules, e.g., an AIE exposure module 
112, an AIE color balance module 122, and AIE contrast module 132, and AIE 
luminance module 182, and an AIE sharpness module 192, by way of buses 110, 120, 
130, 180, 190, respectively. Those of ordinary skill in the art will recognize that other 
AIE modules can form a part of the AIE unit 50, and it is not intended that the invention 
be limited to the particular AIE modules shown. The AIE modules can also return 
information to the information channel 108 for use by other AIE modules. The AIE 



modules 112, 122, 132, 182, 192, in the AIE unit 50 are described in the aforementioned 
Eschbach et al. and Fuss et al. U.S. Patents in greater detail. 

[0044] Each of the exposure, color balance, contrast, and luminance AIE 

modules 112, 122, 132, 182 use the data supplied on the information channel 108 to 
calculate the input signals for the respective TRC generators 114, 124, 134, 184 that 
generate intermediate TRC's required to effect the required image modifications as 
determined by each module 112, 122, 132, 182. Each of the TRC generators 114, 124, 
134, 184, directs a generated intermediate TRC to a TRC combiner 140 that combines 
the respective intermediate TRC's into a single, final TRC 160 defined by a look-up 
table (LUT). The final TRC is then directed to the bind TRC unit 150 of the TRC/filter 
binder 52 wherein it is bound to the JPEG 2000 compressed image data for subsequent 
use. Similarly, a sharpness module 192 receives the low-pass subband LL from the Mth- 
level from the information channel 108 and uses same to generate a sharpness filter 194 
that is also directed to the TRC/filter binder 52, in particular to a bind filter unit 200 
thereof, wherein the filter is bound to the compressed data (along with any TRC that has 
already been bound to the compressed data by the bind TRC unit 150. The combined 
TRC/Filter and JPEG 2000 compressed image data is output at 250 to downstream 
image processing modules 66 (as best seen in Figure 6). It should be recognized by those 
of ordinary skill in the art that the final TRC 160 and/or sharpness filter 194 can be 
bound or otherwise operatively associated with the JPEG 2000 compressed image data 
for any desired length of time, including a fraction of a second or months, years, or 
longer such as for long-term archival storage. Furthermore, the final TRC 160 and/or 
filter 194 can be used immediately once derived, assuming the compressed image data 
has been suitably decompressed. 

[0045] Those of ordinary skill in the art will recognize that the low-pass subband 

LL from the Mth-level extracted from the bound data provide a subsampling or lower 
resolution variant of the uncompressed input image data, without requiring that the 
image data be entirely decompressed. Thus, the low-pass subband LL from the Mth- 
level can be used by the AIE systems described above without significant modifications 
to these systems. Moreover, selectively changing which level low pass bus-band is used 
can correspondingly change the accuracy of the enhancement parameters calculated. 
[0046] The invention has been described with reference to preferred 

embodiments. Modifications and alterations will occur to others upon reading and 
understanding the preceding specification. It is intended that the invention be construed 



as including all such modifications and alterations insofar as they fall within the scope of 
the appended claims or equivalents thereof. 
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